Technical Field
The present invention is directed to methods and systems for generating, rendering, manipulating and communicating ink data that reproduces a path of hand-drawn (freehand) stroke data and renders the path with style.
Description of the Related Art
Various handwriting input systems are known, which allow a user to input hand-drawn (or freehand) data by using a pen-shaped device. For example, electromagnetic resonance type pen-tablet input systems are known, which allow input of hand-drawn data including associated pen pressure and pen tilt data. As further examples, electrostatic type pen input systems are known, which generate capacitance between an implement and a (tablet) sensor surface similarly to how capacitance is created between a finger and the sensor surface. Still further, input systems that output relatively simple information such as gesture information derived from a collection of determined positions are also known.
Typically, hand-drawn data or stroke (path or trace) data inputted by a pen-shaped implement is usable in a single drawing application to generate raster data such as pixel data or image data. A need exists for methods and systems that permit hand-drawn data or stroke data generated by operating a variety of types of devices and applications, such as ink messaging, ink archiving and retrieval applications, e-mail, photo annotation, remote video conferencing applications, etc., to be shared amongst various devices. Digital ink or ink data (hereinafter “ink data”) is proposed to address such need. Typically raster data such as direct pixel data or image data is used, which is generated according to the setting of a particular application used to support a user's stroke input operation on an input device. The ink data, on the other hand, is intermediate data, which exists prior to rasterization of stroke data and which is in the form of vector data usable by a variety of applications. Sample ink data types are described in the following non-patent literature DOCUMENTS (D1) through (D4):    (D1) W3C, Recommendation 20, September 2011, “Ink Markup Language (InkML)” (URL—http://www.w3.org/TR/2011/REC-InkML-20110920/)    (D2) Microsoft Corporation, et al., “Ink Serialized Format Specification” 2007 (URL—http//download.microsoft.com/download/0/B/E/0BE8BDD7-E5E8-422A-ABFD-4342ED7AD886/InkSerializedFormat(ISF)Specification.pdf)    (D3) W3C Working Draft 11, February 2014, “Scalable Vector Graphics (SVG) 2” (URL—http://www.w3.org/TR/SVG2/); W3C Recommendation, 16 Aug. 2011, “Scalable Vector Graphics (SVG) 1.1 (Second Edition)” (URL—http://www.w3.org/TR/2011/REC-SVG11-201110816/)    (D4) W3C, “HTML5 A vocabulary and associated APIs for HTML and XHTML W3C Recommendation 28 Oct. 2014” (URL—http://www.w3.org/TR/html5/)    (D5) Slate Corporation, et al., “JOT—A Specification for an Ink Storage and Interchange Format”, Version 1.0, September 1996
Briefly, the InkML (D1) and ISF (D2) data structures represent stroke data inputted by a pen-type device in a manner sharable amongst different applications. SVG (D3) provides a Web standard that permits drawing of a path defined by user-input control points as vector data, regardless of what type of pen device is used as an input device.
The ink data described in (D1) through (D4) all define geometric information needed to reproduce a trace (or path) formed by movement of a pen or a finger. Such information is herein collectively called a “stroke object.”
(D1) describes the ink data that is currently most widely known. (D1) defines an object called “trace” as follows: “<trace> is the basic element used to record the trajectory of a pen as the user writes digital ink.”
For example,
<ink><trace>x1 y1, x2 y2, . . . xn yn</trace></ink>
describes a path of a stroke object that extends from a point x1, y1 to a point xn, yn.
(D2) describes the ink data generated by an ink function usable on Microsoft™ Windows™ applications. (D2) defines an object called “stroke” as follows: “As described earlier in the simple example, Strokes are the most fundamental and important property in ISF. Strokes contain the packet data that make up the individual points in a stroke and potentially other per-stroke properties as well.”
(D3) describes a standard of a vector data supported by various browsers and drawing software, though (D3) does not assume pen input. (D3) defines information called “path” as follows: “Paths represent the outline of a shape which can be filled, stroked, used as a clipping path or any combination of the three.” In SVG (D3), a path object is interpolated based on interpolation curves such as the Poly-Bezier (Cubic Bezier, Quadratic Bezier) Curves well known in the art.
For example,
<path stroke=“green” stroke-width=“5” d=“M100,200 C100,100 300,100 300,200”/>
describes a path starting from a beginning control point (100,200) to an ending control point (300,200), using two control points (100,100) and (300,100), and having a path width of “5” and color green.
(D4) defines a class called “Canvas Path,” which can utilize, for example, a Quadratic Curve command and a Bezier Curve command to generate interpolated curves.
In the present description, the term “stroke object” is used as a general term that encompasses the “trace,” “stroke,” “path” and “Canvas Path” of (D1) through (D4) above.
A stroke object is vector data information whose data structure includes a set of point or control point coordinates that are used collectively to reproduce a trace (or a path) formed by movement of a pen or a finger. According to various embodiments, the present invention offers methods and systems for generating, manipulating (e.g., slicing), rendering and communicating ink data that represent hand-drawn (freehand) stroke data on and between various applications. Each of the embodiments provide technical solutions that were not available in the prior art of (D1)-(D5) above. It should be noted that, while the following description is organized to disclose generally four (4) embodiments of the invention, various aspects of the embodiments may be combined, supplemented, interchanged, switched or modified among and between the embodiments to produce further embodiments, as will be apparent to those skilled in the art. For example, various methods and systems of each embodiment may employ the definition of ink data, as well as the methods of generating, reproducing, drawing (rendering), manipulating and communicating the ink data and the ink data structures (data objects and data formats) as described in connection with one or more of the other embodiments disclosed herein.
Each of the following embodiments 1-4, in various examples, addresses one or more of the aspects described below.
[ASPECT ONE] Introduction of manipulation objects that partially or wholly transform pre-existing stroke objects in several computers.
According to one aspect, the invention is directed to providing manipulation objects. The previously known ink data models described above include semantics and syntax usable only for processing static stroke data, to process one stroke object as one aggregate. Thus, the previously known ink data models are not capable of selecting or slicing a portion of a stroke object. Also, the previously known ink data models allow manipulation of a stroke object on one processor, and are incapable of allowing multiple processors to share the manipulation (e.g., editing) operation executed on the stroke object in real time.
FIG. 91 illustrates an example of a manipulation object 270, a “slice” object, according to an embodiment of the present invention. A slice object 274 capable of manipulating (slicing) a portion of a stroke object is generated and transmitted. In the illustrated example, a portion of one stroke object 9101 on one computer is sliced, and a manipulation data 9103 indicative of the sliced portion is shared by other computers such that the stroke object 9101 on the other computers too can be manipulated in the same manner. Modification or manipulation (e.g., slicing) of a portion of a stroke object will be described in detail below in the first and fourth embodiments of the present invention. Sharing of one manipulation object 270 amongst multiple computers to share the edited, up-to-date status of the ink data among them will be described in detail below in the first, second and fourth embodiments of the present invention.
[ASPECT TWO] Abstracting the definition of pen event input information to absorb device differences (and making SVG more pen-input-oriented to improve SVG's pent-input expression capability).
According to a further aspect, the invention is directed to making hand-drawn input data abstract so as to absorb any differences that exist among different input devices. This is achieved by abstracting pre-existing input attributes of strokes, such as pen pressure and pen angle information, to higher-level-concept attributes defined in a novel model. In general, the information that needs to be reproduced based on hand-drawn input data is not “how” the hand-drawn data was inputted, such as at what angle a pen (stylus) was held, at what point in time what coordinate was obtained, and how much pen pressure was applied, etc. Instead, the information that needs to be captured is vector data that can reproduce the “result” of such pen (style) operation or drawing operation that was carried out with certain pen pressure, pen speed, etc.
Currently various hand-drawn input devices exist, ranging from a high-performance input device (e.g., 9202C in FIG. 92) capable of obtaining pen pressure, pen angle, pen rotational angle data, etc., to a widely used electrostatic tablet or other simpler input devices capable of receiving input by a finger but not capable of obtaining pen pressure, pen tilt angle, etc. (e.g., 9202A in FIG. 92). Thus, it is desirable to convert any device-dependent attributes of hand-drawn input data (shown as “Device dependent Pen Event Data” of 9202A-9202C in FIG. 92, for example) to device-independent abstracted vector data (9204 in FIG. 92), which can be used to reproduce the “result” of a pen event. The ink data defined in such an abstracted form may be organized in vector data, to ultimately produce raster data (image data) as shown in 9208 in FIG. 92. SVG11 (D3) discussed above defines vector data, and is shown as 9206 in FIG. 92. SVG11 (D3) does not permit varying or adjusting the stroke width, color and transparency (opacity) and, as a result, is not particularly suited for reproducing the “result” of a pen event. Also, SVG includes data other than the stroke object path coordinates data, such as control points used to generate Bezier curves, and thus are not suited for use with various applications 9220 other than specialized drawing applications.
In addition to producing raster image data (9208, FIG. 92), it is also desirable to organize the ink data in a more abstracted form in vector, for use in a signature verification application, in an annotation application, etc. In this regard, abstraction is preferably not too image-oriented, but should result in abstract attributes that may be used to define ink data in both raster form and in vector form. Abstracting device-dependent pen event data 9202 of Type 1 (including pen pressure data) and of Type 2 (not including pen pressure data) to the generalized ink data, which is the intermediate data 9204 in FIG. 92, will be described in detail below in the first and third embodiments of the present invention.
[ASPECT THREE] Extending the life cycle of an ink data ecosystem by separating a language (information model) from a format.
For example, contents of raster data such as digital photos are often used not only by a single service or on a single application, but by multiple services and applications and are shared by or transferred amongst all in a chained manner on a particular “ecosystem” (though they may be processed in various formats such as JPEG, GIF, TIFF, etc.). These various formats may be used because raster data includes a common information model which conceptually describes a collection of pixel values.
According to a still further aspect, the invention is directed to facilitating ink data exchange and transfer between different formats, based on adoption of the common language (stroke language (SL)). The stroke language (SL) is an information model that defines semantics of the ink data of the present invention, as opposed to the formats of the ink data. That is, the ink data thus defined by abstracted attributes may be processed into different raster image formats (PNG, JPEG, etc.), exchanged between different vector graphics formats (SVG, InkML, HTML5, etc.), or produced in different stream formats (ISF, InkML, etc.) that define stroke structures. FIG. 93 conceptually describes this aspect of the invention. To add flexibility to output format types as well as input format types, and to accommodate a variety of output and input format types, the common language (or the information model that defines the common language) preferably resides in the intermediary between a device driver level that generates the language and an output level at which the generated language is outputted into a file, packets, etc. In particular, the ink data processing section 100 according to various embodiments of the invention includes an ink data generation section 120 that generates ink data based on the abstracted language (stroke language), and an ink data formatting section 140 that handles input and output of the ink data in various formats, as two separate components. Since the function of ink data generation and the function of ink data formatting for input/output purposes are separated, the ink data processing section 100 is suited to be used as a building block of the ink data ecosystem to spread use of the ink data amongst various devices. This aspect of the invention will be described in detail below in the fourth embodiment.
These three aspects of the invention as described in FIGS. 91-93 will be discussed again after the description of the first through fourth embodiments of the present invention below.